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The  Johnson-Neyman  technique  is  a  statistical  tool  used 
most  frequently  in  educational  and  psychological  applica¬ 
tions.  This  paper  starts  by  briefly  revieving  the  Johnson- 
Neyman  technique  and  suggesting  when  it  should  and  should 
not  be  used;  then  several  different  modifications  and  ex¬ 
tensions  of  the  Johnson-Neyman  technique,  all  of  them  con¬ 
ceptually  simple,  are  proposed.  The  close  relation  between 
confidence  intervals  and  regions  of  significance  of  the 
Johnson-Neyman  type  is  pointed  out.  The  problem  of  vh.  t 
to  do  when  more  than  two  groups  are  being  compared  is  con¬ 
sidered.  The  situation  of  more  than  one  criterion  variable 
is  also  conridered. 
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ON  THE  JOHNSON-NEIMAN  TECHNIQUE  AND  SOME 


EXTENSIONS  THEREOF1 
by 

Richard  F.  Potthoff 
University  of  North  Carolina 

1.  SUMMARY  AND  INTRODUCTION 

When  measurements  on  one  criterion  variable  and  two  or  more  predictor 
variables  are  available  for  the  members  of  two  groups  (groups  of  classes  or  of 
individuals,  e.g.),  the  Johnson-Neyman  technique  £5,  4,  l7  may  used  t°  obtain 
a  region  of  values  of  the  predictor  variables  for  which  the  null  hypothesis  of  no 
difference  in  expected  criterion  score  between  the  two  groups  would  be  rejected 
at  the  .C5  level  (or  at  the  a  level,  if  we  want  to  be  more  general).  Thus,  for 
any  specific  point  P  within  this  region  of  significance  determined  by  the 
Johnson-Neyman  technique,  the  number  0  does  not  lie  within  the  95°/°  confidence 
interval  for  the  difference  in  expected  criterion  score  between  the  two  groups 
at  point  P  ,  and  hence  one  can  be  at  least  95°/o  confident  that  there  is  a  true 
difference  between  the  two  groups  at  point  pj  however,  it  does  not  follow  that 
one  can  state  with  95 °/o  confidence  that  there  is  a  non-zero  difference  between 
the  two  groups  simultaneously  for  all  points  in  the  region.  A  statement  of  this 
latter  type  is  related  to  simultaneous  confidence  bounds:  if  the  experimenter 
wants  a  region  atout  which  this  sort  of  statement  can  be  made,  then  a  modification 
of  the  Johnson-Neyman  technique  is  required,  as  will  be  described  in  Section  3. 


^This  research  was  supported  in  part  by  Educational  Testing  Service,  and 
in  part  by  the  Mathematics  Division  of  the  Air  Force  Office  of  Scientific  Research. 
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An  approach  which  is  essentially  similar  to  the  region  approach,  but  which  uses 
simultaneous  confidence  intervals  in  lieu  of  getting  a  region  plotted,  is  dis¬ 
cussed  in  Section  h. 

The  Johns on-Neyman  technique  was  designed  for  a  situation  in  which  there 
are  just  two  groups  being  compared.  To  handle  the  case  where  there  are  more  than 
two  groups,  Section  5  develops  a  simple  technique  which  is  based  on  simultaneous 
confidence  bounds  for  the  differences  between  all  possible  pairs  of  groups. 

Another  extension  of  the  Johns on-Neyman  technique  is  dealt  with  in  Section 
6,  where  the  case  of  more  than  one  criterion  variable  is  considered.  Again  the 
approach  is  simply  that  of  simultaneous  confidence  bounds,  but  different  tools 
are  used  this  time. 

The  techniques  suggested  in  this  paper  all  require  the  calculation  of  many 
of  the  same  quantities  which  must  be  computed  for  the  Johns on-Neyman  technique; 
beyond  this  point,  the  remaining  calculations  required  are  of  an  elementary 
nature  in  all  of  the  techniques. 

To  start  off.  Section  2  will  briefly  review  the  foundations  of  the  Johnson- 
Neyman  technique  itself,  and  will  indicate  how  the  technique  should  and  should 
not  be  used" 


2.  THE  JOHNSON -NEYMAN  TECHNIQUE 

We  suppose  that  we  have  two  groups  of  classes  or  of  individuals,  with 
members  in  the  first  greup  and  members  in  the  second  group.  Each  member  of 
each  group  is  measured  cn  each  of  r  predictor  (control)  variables 


X,,  ...,  X  :  let 

id  r 


Xijk  den0‘te  the  measurement  for  the  k-th  member  of  the 


on 
•"  □ 
□ 


j-th  group  on  the  i-th  predictor  variable  (k  =  1,  2,  . . . ,  n  ;  j  =  1,  2;  i  ■=  1,  2, 

J 

r).  Each  member  of  each  group  is  also  measured  on  the  single  criterion 
variable,  and  we  will  let  Y  denote  this  criterion  measurement  for  the  k-th  - 
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member  of  the  j-th  group,  (if  the  members  of  the  groups  are  classes  rather 

than  individual  students,  then  the  YJk’s  and  the  X^’s  would  of  course 

be  class  average  scores  rather  than  individual  student  scores).  In  the  model 

we  assume  that  the  Y  's  are  normally,  independently,  and  homoscedastically 

JK 

distributed,  and  that  the  expectation  of  Y.,  for  given  X's  is  of  the  form 

JK 


(2.i)  E(YJk)  -  aj  ♦  X1Jk  ♦  b2J  X2jk  +  ...  ♦  brj  X^ 


(k  =  1,  2,  ...,  ry*  j  =  1,  2) 


where  the  a.' 3  and  the  b  's  are  unknown  regression  parameters.  /“Some  kind 
J  J 

of  attempt  should  be  cade  to  check  that  the  homos c edas tic i ty  assumption  is  at 
least  roughly  satisfied.  Two  ways  in  which  unequal  variances  could  easily  arise 
are  (a)  the  variances  night  be  different  for  the  two  groups,  i.e.,  the  Y^fs 
might  have  a  variance  different  from  the  Y^/s;  and  (b)  if  the  different  k-values 
represent  different  classes  of  students,  then  radical  differences  in  the  class 
sizes  might  cause  appreciable  heteroscedasticity/7 

In  the  original  paper  /"* 57  >  the  Johns on-Neyman  technique  was  formulated 
for  r  =  2  predictor  variables.  However,  it  has  been  extended  explicitly  (see 
Abelson  /~ ij ,  e.g.)  to  cover  the  case  where  there  is  a  general  number  (r)  of 
predictor  variables,  as  in  the  model  (2.1).  The  essence  of  this  development  of 
the  Johnson-Neyisan  technique  for  general  r  may  be  summarized  as  follows.  Let 
Cj  (j  -  1,  2)  be  an  rxr  matrix  whose  general  element  is 


k^  Xijk  XIjk 


( j!  w 


(2.2) 


b 


Let  u  (j  =  1,  2)  be  an  rxl  vector  whose  i-th  element  is 
J 


(2.3)  u. 


n. 

. ,  =  ZJ  X...  Y.. 

ij  k=1  ijk  jk 


(  X  )  (  ZJ  Y..  ) 

k=l  lJK  k=l 


Then  we  write  the  rxl  vector 


AAA  A  » 

(2.4)  b.  =  (b^,  b 2y  brj) 


C-*1  u 
i  j 


for  the  estimates  of  the  h-coefficients  for  j  =  1,  2.  The  error  sum  of  squares 


2  2  n  2  Y$s) 

(2.5)  =  L  Zj  Y*  -  Z  - 

j=l  k=l  JK  j=l  j 


2  r  A 
j=i  i=i  1J  0 


2  2 

and.  the  error  mean  square  is  =  Sg/f  ,  where 


(2.6) 


f  =  Z  (n.  -  r  -  1) 
J=1  3 


Let  X,  (j  =  1,2)  denote  an  rxl  vector  whose  i-th  element  is 

ni 

£  xiJk 

(2-7>  \i  •  ;bhrp~ 

Also  we  define 


(2.8) 


ni 

2  Yv< 

k=l  0 


liow  let  the  rxl  vector 


(2.9) 


X  =  (x1#  x2,  ...,  xr) 
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denote  a  set  of  values  of  the  control  variables.  We  would  reject  at  the  a 


level  the  hypothesis  of  no  difference  in  expected  criterion  score  between  the 
two  groups  at  the  point  X  (2.9)  if 


(2.10) 


(Y2  -  b»  ^)  -  (Yx  -  b»  X1)  +  (b|  -  bp  X _ 


>  t 


f;a 


where  t^  is  the  (l  -  ^f)  fracxile  of  the  t  distribution  with  f(2.6)  degrees 
of  freedom.  The  inequality  (2.10)  defines  a  region  (which  we  will  call  R)  in  the 
r- dimensional  space  of  the  predictor  variables;  it  is  this  region  which  ia  .the 
Johnson-Neyran  region  of  significance.  We  may  re-write  (2.10)  in  the  form 


(2.11)  -  b >  Xg  -  +  bj  Xx)  +  (b'  -  b£)  X_72 


*  *f;a  Se 


/*-“  +  b  +  (x’-x{)c"1(x-x1)  +  (X’-X’jc-Vxgiy  >  0  ; 


actually,  there  are  still  other  ways  of  writing  the  inequality  which  defines  R 
(see  fl 7,  e.g.). 

The  Johnson-Neyman  region  of  significance  can  be  plotted  when  r  =  2,  but  the 
situation  l sccmes  much  more  troublesome  when  r  >  2.  Because  of  this  difficulty 
in  plotting  R  when  r  >  2,  the  usefulness  of  the  Johns on-Neymen  technique  for 
general  r  has  been  questioned. 

There  is  a  second  possible  trouble  point  regarding  the  Johnson-Neyman  techni¬ 
que  which  has  perhaps  never  been  sufficiently  emphasized.  As  indicated  in  Section 
1 ,  we  can  be  at  least  95°/o  confident  (for  a  =  .05)  that  the  two  groups  are 
different  for  any  specified  individual  point  in  R,  but  we  cannot  be  95°/o 
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confident  that  the  two  groups  are  different  simultaneously  for  all  points  in  R. 

In  seme  cases  one  might  desire  a  statement  of  the  former  kind,  in  other  cases 
a  statement  cf  the  latter  kind.  For  instance,  if  one  is  interested  only  in  a 
single  specific  class  of  students  (and  if  the  two  groups,  j  =  1  and  2,  represent 
two  curriculums ,  e.g.),  then  it  would  be  legitimate  to  inquire  whether  the  point 
X  (the  set  of  control  variable  scores)  for  the  class  falls  within  the  region  R, 
in  order  to  determine  whether  the  two  curriculums  may  be  assumed  to  have  different 
effects  for  that  class;  however,  for  a  purpose  of  this  sort,  it  would  appear  to 
be  not  only  less  difficult  but  also  more  informative  to  obtain  a  confidence  inter¬ 
val  for  the  true  difference  in  effects  between  the  two  curriculums  for  that 
class,  rather  than  to  urdertake  the  labor  of  plotting  R.  On  the  other  hand,  if 
one  is  interested  in  nuking  a  95°/o  confidence  statement  about  an  entire  point- 
set  in  the  X-space  (as  would  likely  be  the  case  if  one  were  interested  in  over¬ 
all  educational  policy  for  many  classes,  zsther  than  in  a  single  c3ass),  then  it 
would  not  be  proper  to  use  the  Johns on-Neyman  region  R  since  the  simultaneous 
confidence  coefficient  for  such  a  region  would  generally  be  under  95°/o. 

All  of  this  suggests  that  perhaps  some  small  modification  or  extension  of  the 
idea  behind  the  Johnson-Neyman  technique  might  be  appropriate.  In  the  next  two 
sections  we  will  suggest  two  possible  approaches  along  these  lines.  First,  Section 
3  will  indicate  one  technique  for  obtaining  a  region  (point-set)  whose  simultaneous 
confidence  coefficient  is  >  95°/o.  Second,  Section  U  will  present  a  confidence 
interval  approach  which,  in  a  certain  sense  furnishes  all  the  information  which 
the  region  approach  furnishes  plus  some  more  but  which  does  not  provide  us  with 
a  region  to  be  plotted;  actually,  a  region  approach,  if  interpreted  in  a  certain 
way,  is  practically  equivalent  to  a  group  of  confidence  statements  anyway. 
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3.  A  "SD'IU  LTANEOUS "  REGIOE 

In  this  section  we  will  present  one  technique  for  obtaining  what  we  will 
call  a  "simultaneous"  region  of  significance,  i.e. ,  a  region  (which  we  will  call 
R')  such  that,  with  confidence  >  95°/o  >  we  can  state  tl»t  the  two  groups 
(j  s  1  and  2)  are  different  simultaneously  for  all  points  in  R' .  In  other 
words,  in  the  long  run,  not  more  than  5°/°  of  such  regions  R'  which  are 
calculated  will  contain  any  points  at  all  for  which  the  two  groups  are  equal  in 
expected  criterion  score. 

Probably  a  number  of  different  techniques  for  obtaining  a  simultaneous  region 
could  be  devised.  Computationally,  the  technique  to  be  presented  here  bears  ar 
extremely  close  resemblance  to  the  Johns on- Neyman  technique,  and  for' this  reason 
is  probably  computationally  simpler  than  most  alternative  techniques  would  be. 

On  the  other  hand,  it  is  possible  that  some  alternative  technique  might  produce 
a  region  which  would  be  "larger"  (in  some  sense)  and  therefore  better  in  that 
respect . 

Our  approach  will  utilize  a  simple  argument  based  on  the  simultaneous  confi¬ 
dence  bound  method  developed  by  Roy  and  Bose  £ 7 >  Section  2.1,  formula  (2.1.6) 
especially7.  Let  us  define 

(3.1)  d21  =  d21(X)  =  (a2  -  +  (b*  -  b')X 

where  b^  (j  =  1,  2)  is  an  rxl  vector  containing  the  r  b^’s.  We  my  write 

A  A  ,  ,A  A  A  A 

(3.2)  d21  =  d21(X)  =  (a2  -  +  (b^  -  b|)  X 

where 


(5.3) 


A 


for  j  =  1,  2.  Also  we  define 


(5.M  Vj.v^tx)  ,  -i-  ♦  (*-  -  Ij)  c]1  (x  - 

for  j  s  1,  2.  Nov  we  are  ready  to  utilize  £'1,  formula  (2. 1.6)7  to  establish 
that  100(l-a)°/o  simultaneous  confidence  intervals  for  the  functions  d2^(X) 
for  all  possible  points  X  in  the  r- dimensional  X- space  are  given  by 


(3-5)  d21(x)  1  y'(r+1)Fr+ijf.a  y/v2^x)  +  vi(x)  "/^e”  ' 

where  F  .  .  denotes  the  (l-a)  fractile  of  the  F  distribution  with  r+1 
r+1,  T  ,  Of 

and  f  degrees  of  freedom.  If  we  examine  (3«5)j  ve  can  conclude  that,  with 
simultaneous  confidence  coefficient  >  100  (l-a)°/o,  we  can  state  that  d^(X)  is 
^  0  (i.e.,  that  the  two  groups  are  different)  for  all  points  X  such  that 


(3.6)  /*d21(xj 72  >  (r+1)  Pr+1>f;a  £ Vg(X)  +  v1(x)J  s* 

Thus  (3.6)  defines  the  simultaneous  region  Rf, 

Note  that  the  defining  inequality  for  the  simultaneous  region  Rf  (3*6)  is 

basically  identical  with  the  defining  inequality  for  the  Johnson-Neyman  region 

R  (2.11),  with  the  one  exception  that  t_.  in  (2.11)  is  replaced  by 

1  >ot 

(r+l)F  ,  .  in  (3*6).  Hence  xhe  computational  procedure  for  obtaining  R*  is 

r+1  yLyOt 

essentially  the  same  as  that  for  obtaining  the  Johns on-Neynan  region  R. 

As  is  the  case  with  the  Jcfanson-Neyman  region,  the  region  R'  will  consist 
of  two  parts:  one  part  where  the  statement  is  trade  that  d^(X)  >  nnd  another 
part  containing  those  X's  for  which  we  state  that  d^(X)  <  0.  Recognizing 
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these  two  separate  parts  is  thus  more  explicit  than  merely  stating  that 
d?1(X)  ^  0  throughout  R'.  As  with  the  Johnson-IIeyman  region,  one  or  both  of 
these  two  parts  of  R'  may  in  some  cases  be  null  (i.e.,  contain  no  points  at 
all). 

It  appears  that  the  "simultaneous"  region  R'  (3.6}  will  always  be  a  (proper) 
subset  of  the  Johnsor.-Neynan  region  R  (2.11)  (i.e.,  every  point  in  R’  will 
also  be  in  R,  but  not  every  point  in  R  will  be  in  Rr).  Thus  R'  could  turn 
out  to  be  "smaller"  than  we  might  like  it  to  be.  However,  it  has  been  pointed 
out  (see  Scheff6  /~8,  p.  71  ~J >  e.g.)  that,  when  we  are  dealing  with  simultaneous 
confidence  bounds,  it  may  be  most  sensible  to  choose  an  at  value  somewhat 
larger  than  what  we  would  customarily  choose  for  simple  confidence  bounds.  Hence 
the  choice  of  a  larger- than- usual  a  would  result  in  R'  not  being  so  "small". 

As  already  indicated,  the  region  R '  (3.6)  may  not  be  the  most  desirable 
100  (l-a)°/o  simultaneous  region  which  can  be  found:  if  we  examine  more  closely 
the  situation  for  r  =  1,  we  will  suspect  that,  for  general  r,  it  should  some¬ 
how  be  possible  for  the  region  R*  (3*6)  to  be  improved  upon  (although  such  im¬ 
provement  might  well  be  at  the  expense  of  increased  computational  difficulty). 

For  r  »  1,  if  we  write  X1Q  =  -  (a 2  -  for  the  value.- of  X1  for 

which  d^X)  /"i.e.,  d21^Xl^7  is  0  (we  assume  b12  j  b^),  then  a  100(l-a)°/o 
confidence  region  for  X^  (see  Fisher  /~3,  ?P«  lMf-l467,  e.g.)  is  specified  by 
the  inequality 


(3-7)  r  d21(x10)_7£ 


5  tf;a 


f 


v2(V  +  VX10}-7  9e 


If  we  state  that  we  are  100  (l-a)  °/o  confident  that  X^q  lies  in  the  region 
(3.7),  this  is  equivalent  to  stating  that  we  are  ICO  (l-a)  °/o  confident  that 
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X1Q  does  not  lie  in  the  complement  of  (3-7)>  which  (in  turn)  is  the  same  thing 
as  saying  that  we  are  100(l-a)°/o  confident  that  dg^(X)  ^  ®  for  X  such 

that 

(3.8)  /"d21(x)_72  >  t2.a  r v2(x)  +  vx(X)_7  S2 

But  we  note  that  (3.8)  is  the  same  as  (3*6),  except  that  instead  of  2F^ 
in  (3*6)  we  have  the  apparently  smaller  quantity  -  in  (3*8).  Hence  R* 

-Lj*  )Ct 

(3-6)  can  evidently  he  improved  upon  for  r=l,  and  although  unfortunately  no 

ready  generalization  of  (3.8)  for  r>.l7”suc^  as  an  inequality  like  (3-8)  hut  with 
2 

t-  replaced  hy  r  F  ..  for  general  r,  e.g.7  could  he  discovered,  one  would 
*  >ct  r,i;o:  ’** 

still  suspect  from  this  that  Rf  (3*6)  could  be  improved  upon  for  r  >  1  also, 

^/~We  take  note  finally,  however,  of  a  curious  fact.  For  the  region  (3*6),  we 

can  state  with  >  100(l-a)°/o  confidence  not  only  that  d^(X)  4  0  for  all 

X  satisfying  (3*6),  but  more  specifically  that  d^X)  >  0  for  all  X  satisfying 

(3,6)  for  which  d^X)  >  0  and  that  d^X)  <  0  for  all  X  satisfying  (3*6) 

A 

for  which  don(x)  <  0#  For  the  region  (3*8),  however,  we  apparently  can  state 

Cjl 

with  100(1 -a) °/o  confidence  only  that  d21(X)  4  0  for  all  X  satisfying  (3-8), 
and  we  cannot  make  any  more  specific  statement  about  the  sign  of  dg^(X)  with¬ 
out  causing  the  confidence  coefficient  to  drop  below  100(3.-cr)o/o.  This  would  he 
a  drawback  of  using  the  region  (3.8 )J 

4.  SIMULTANEOUS  CONFIDENCE  INTERVALS 

The  Johnson-Neyman  approach  and  the  simultaneous  region  approach  which  was 
suggested  in  Section  3  both  require  that  a  region  be  obtained  and  usually  plotted. 
As  already  indicated,  this  nay  not  he  an  easy  task,  especially  when  r  >  2.  There¬ 
fore  it  is  appropriate  to  consider  an  alternative  approach. 
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In  a  certain  sense,  the  confidence  interval  approach  wMs&  we  are  about 
to  suggest  is  almost  equivalent  to  the  region  approach  anyvay;  for  both  the 
region  F  (2.11)  and  the  region  R'  (5-6)  are  essentially  based  on  confidence 
bounds  to  begin  with  (the  former  on  simple  confidence  bounds,  the  latter  on  simul¬ 
taneous  confidence  bounds).  The  confidence  bounds  associated  with  the  region  R 
(2.11)  are  simple  confidence  bounds  on  the  function  d^^Cx)  for  a  specified  X: 
for  any  individual  X,  we  can  be  100(l-a)°/o  confident  that  d^X)  lies  with¬ 
in  the  interval 

(4.1)  4si(x)  i  tf;o  yVg(X)  7  TjW"  /af 

The  relation  between  the  Johns on-Neyman  region  R  (2.11)  and  the  confidence,  inter¬ 
val  formula  (4.1)  is  simply  this:  the  number  0  will  fall  outside  of  the  inter¬ 
val  (4.1)  if  and  only  if  X  lies  in  the  region  R  (2.11)  . 

The  confidence  bounds  associated  with  the  region  R*  (3-61  were  actually 
U3ed  as  a  step  in  the  argument  which  developed  (3*5),  and  are  specified  by  the 
simultaneous  confidence  interval  formula  (3*5)*  The  number  0  will  fall  outside 
of  the  interval  (3*5)  if  and  only  if  X  lies  in  the  region  R’  (3*6)* 

What  we  trill  now  propose  is  that,  in  some  situations,  it  may  be  more  sensi¬ 
ble  to  utilize  the  confidence  intervals  (4.1)  or  (3*5)  in  lieu  of  obtaining  the 
regions  (2.11)  or  (3-6)  respectively.  We  have  already  notelthe  close  relation 
between  the  confidence  intervals  and  the  regions.  In  one  respect,  the  confidence 
interval  approach  actually  gives  more  information  than  the  region  approach:  the 
former  furnishes  us  with  a  specific  confidence  interval  for  ^or  an^ 

whereas  the  latter  does  not  provide  us  with  the  interval  but  rather  tells  us  only 
(for  every  X)  whether  or  not  this  confidence  interval  contains  the  value  0.  On 
the  other  hand,  though,  the  confidence  interval  approach  does  not  furnish  us 
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explicitly  wit  -  .ny  graphical  "region  of  significance". 

The  simp'e  confidence  interval  (4.1),  which  is  associated  with  the  Johnson- 
Neyman  region  R  (2.11),  may  be  used  (as  was  already  indicated  in  Section  2) 
when  we  want  confidence  bounds  on  d^(X)  for  a  single  specific  X.  The  use  of 
the  confidence  interval  approach  with  respect  to  the  simultaneous  bounds  (3*5)> 
however,  requires  somewhat  more  discussion  than  the  formula  (4.1)  because  there 
are  different  ways  in  which  (3*5)  can  be  utilized;  we  mention  some  possibilities: 

(a)  Tb*'  intervals  (3*5)  could  be  calculated  for  a  certain  number  of  strate¬ 
gically  -  placed  points  in  the  X- space.  For  example,  a  certain  number  of  equally- 
spaced  values  could  be  chosen  for  each  one  of  the  r  control  variables,  and 
(3.5)  could  then  be  calculated  for  all  possible  combinations  of  these  values, 
thereby  furnishing  us  with  the  intervals  (3-5)  for  all  the  points  on  a  sort  of 
r-dimensional  grid  in  the  X- space. 

(b)  We  might  learn  something  interesting  by  calculating  (3-5)  for  the 

(n^  +  n2)  points  in  the  X- space  represented  by  the  •  •  •  '*rik^,S 

(k  =  1,2,...,  nx)  and  the  (X^,  X^, . . .  tX^)  fs  (k  »  1,2,...,  ng).  The  idea 
would  be  that  these  (n^  +  n^)  points  would  in  many  cases  be  a  reasonable  cross- 
section  of  the  particular  population  of  points  in  the  X-space  in  which  we  would 
be  interested.  Furthermore,  we  might  classify  each  of  the  (n^  +  n^)  intervals  into 
one  cf  three  groups:  intervals  lying  wholly  above  0,  internals  lying  wholly  be¬ 
low  0,  and  intervals  which  contain  0  /"and  the  last  group  could  be  split  into 

A 

two  parts,  if  desired,  according  to  whether  d^(X)  >0  or  <  0/*  It  might  then 
be  instructive  to  tally  the  number  of  intervals  in  each  group. 

(c)  We  might  simply  have  a  specific  list  of  points  in  the  X-space  for  which 
we  wish  to  obtain  simultaneous  confidence  intervals,  where  the  points  might  (e.g.) 
be  the  X-scores  for  a  list  of  certain  classes  (cT  students)  which  we  are  interested 
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in  and  which  need  not  be  among  the  (n^  +  n^)  classes  appearing  in  the  data. 

5.  THE  CASE  OF  MORE  THAN  TWO  GROUPS 

The  methods  we  have  discussed  so  far  have  been  for  situations  where  there 
are  only  two  groups  (j  »  1  and  2).  These  two  groups  nay  represent  (e.g.)  two 
different  curriculums  or  two  different  teaching  techniques.  We  now  consider  a 
more  general  situation  where  there  are  g  groups  (j  =  1,2,..., g)  being  compared 
rather  than  Just  two.  We  will  vise  the  same  notation  and  assumptions  as  before 
f  see  particularly  (2.1  -  2.4),  (2.7  -  2.8),  (3-3  -  3»4Jj7>  tut  with  the  under¬ 
standing  that  the  subscript  j  may  assume  the  values  lr2, ...,g  instead  of  just 
the  values  1  and  2. 

A  total  of  g(g-l)  different  possible  pairs  of  groups  can  be  chosen  from 
among  the  g  groups.  Let  (j,  j)  be  any  such  pair  (we  assume  J  >  J  for 
definiteness).  Then,  generalizing  (3.1),  we  can  define 

(5.D  *  dJj(X)  -  (aj  -  aj)  +  (bj  -  bj)  X  (J  >  J)  . 

Thus,  for  the  g-group  situation,  we  are  potentially  interested  in  the  ^g(g-l) 

differences  d  (5-1)  rather  than  Just  in  a  single  difference  d  ( 3 - 1 ) * 

J  J  “• 

Our  problem  is  to  generalize  the  techniques  available  for  the  two-group 
situation  in  order  to  obtain  ways  to  handle  the  g-group  situation*  The  idea 

suggested  here  will  be  a  simple  one  which  again  will  be  based  on  the  simultaneous 

~  * 

confidence  bound  method  given  by  Roy  and  Bose  fl ,  Section  Z.lJ'.  we  will  obtain 

simultaneous  confidence  bounds  on  all  ^g(g-l)  possible  djj's  . 

If  we  are  interested  in  a  single  specific  X,  then  100(l-a)°/o  simultaneous 

confidence  intervals  for  d,,(X)  for  all  possible  (j,j)  but  for  this  single  X 

J  J 

are  given  by 

(5.2)  a,j(x)  *  yr /v,U)  *  v^x) 


14 


A  o 

where  dT.(X)  is  defined  analogously  to  (3.2),  and  where  S  and  f  are  defined 
Jj  2  € 

the  sane  as  (2*5)  and  (2.6)  respectively  except  with  L  replaced  wherever  it 

g  J-l 

appears  by  2  .  If  we  are  interested  in  simultaneous  hounds  for  all  X,  then 

o  J=1 

lCO(l-a)  /o  simultaneous  confidence  intervals  for  d„,(X)  for  all  possible  (j,j) 

jj 

and  for  all  possible  X  are  given  by 


(5*3)  djj^x)  1  '/^r+1^g"1^F(r+1)(g.1),fJa  VVX)  +  vj* 


2 

where  Sg  and  f  are  defined  the  same  as  for  (5*2).  The  formulas  (5 -2)  and 
(5-3)  are  both  immediate  consequences  of  £ 7>  formula  (2. 1-6)7.  Note  that  (5-2) 
is  a  generalization  of  (4.1),  while  (5-3)  is  a  generalization  of  (3*5)*  Many  of 
the  remarks  about  the  confidence  interval  approach  which  were  imde  in  Section 
4  can  also  be  applied  (with  appropriate  modification)  to  the  confidence  intervals 
(5.2)  and  (5.3). 

If  a  region  approach  is  preferred  to  a  confidence  interval  approach,  it  is 
possible  to  obtain  regions  of  significance  corresponding  to  either  (5*2)  or  (5*3). 
However,  if  all  -|g(g-l)  pairs  (j,  j)  were  considered,  it  would  be  necessary 
/""in  the  case  of  either  (5*2)  or  (5«3J7  to  obtain  -|g(g-l)  different  regions  (i.e., 
one  region  for  each  comparison).  Statements  could  be  made  with  100(l-a)°/o  con¬ 
fidence  with  respect  to  all  ^g(g-l)  regions  simultaneously.  The  regions  corres¬ 
ponding  to  (5*3)j  for  example,  are  based  on  the  inequalities 


(5.M  rdjjttjf  >  (rnXg-l)F(rtl)(g.1))f.a  r VX>  *  Vx)-7 


and  so  (5-4)  is  a  generalization  of  (3-6)  . 


/ 
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6.  THE  CASE  OP  MORE  THAN  ONE  CRITERION  VARIABLE 
So  far  we  have  considered  only  the  case  nf  a  single  criterion  vuriacle, 
However,  situations  are  sometimes  encountered  where  there  are  two  or  nore  criterion 
variables.  For  example,  suppose  that  we  have  an  experiment  with  three  curriculums, 
and  suppose  that  each  curriculum  has  a  different  test  associated  with  it  which  is 
customarily  administered  to  the  students  at  the  end  of  the  course.  In  our  ex¬ 
periment,  however,  suppose  that  all  three  tests  are  administered  at  the  end  of 
the  course  to  all  classes  in  each  of  the  three  groups,  in  order  to  obtain  all  the 
information  which  might  be  necessary  for  a  fair  comparison.  Then  the  three  tests 
would  constitute  three  different  criterion  variables. 

Thus  we  are  faced  with  the  problem  of  extending  the  techniques  described 
earlier  in  the  paper  to  the  case  *f  multiple  criterion  variables.  Our  model 
is  now  a  multivariate  one;  instead  of  (2.1)  we  have  the  model  equation 


(6.1) 


■ 


.(f) 


bW 

U 


ljk 


b(f)x 

b2J  X2jk 


+  b 


(f)  V 

rj  rjk 


where  the  superscript  %  is  the  index  referring  to  the  criterion  variable.  We 

suppose  that  there  are  p  criterion  variables,  so  that  f  =  l,2,...,p.  One 

((f) 

would  ordinarily  assume  for  eech  fixed  (j,k)  that  the  p  Y”/fs  follow  a  multi¬ 
variate  normal  distribution  with  unknown  variance  matrix  E(pxp),  although  for 
the  particular  technique  which  we  are  about  to  propose  it  so  happens  that  it  will 
be  sufficient  to  assume  only  that  the  univariate  marginal  distributions  are  normal 
and  hcmoscedastic  for  each  of  the  p  variates. 

To  attack  the  problem  of  the  multiple  criterion  variables,  we  will  start  by 
supposing  that  we  are  basically  interested  in  obtaining  simultaneous  confidence 
bounds  on  the  differences 


/ 
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(6.2) 


=  <•? 


a  (0)  + 
J 


for  all  (j,  j)  and  for  all  ^  and  either  for  all  X  or  for  a  single  specific 
X.  f  In  (6.2),  is  of  course  a  vector  containing  the  b^p’s  •! 

It  .ould  he  valid  to  apply  Roy’s  formula  for  simultaneous  multivariate  con¬ 
fidence  "bounds  (see  /~6,  p.  101,  formula  (14.6.3J7*  e.g#)  here,  but  some  rough 
investigations  indicate  that  an  alternative  technique  probably  provides  shorter 
confidence  intervals  in  this  particular  case.  Ve  are  interested  in  confidence 
bounds  on  a  type  of  function  (6.2)  which  is  rather  specialized  in  the  sense  that 
any  single  function  (6.2)  involves  but  one  value  of  jf;  and  Roy’s  general  formu- 
la  r 6y  formula  (14.6.3J7  appears  to  be  most  efficient  not  for  such  relatively 
specialized  functions,  but  rather  for  functions  involving  parameters  associated 
with  more  than  one  variate. 

The  technique  of  Roy  and  Bose  JTl,  Section  2.l7  which  we  employed  to  advan¬ 
tage  in  both  Section  4  and  Section  5  cannot,  of  course,  be  appealed  to  again  here, 
because  it  applies  only  to  a  univariate  situation.  The  device  we  will  suggest  for 
getting  simultaneous  bounds  on  the  functions  (6.2)  is  conceptually  a  simple  one, 
and  is  based  on  essentially  the  same  idea  which  was  employed  by  Bunn  /~27:  we 
just  generalize  any  of  our  previous  confidence  interval  formulas  by  substituting 
(a/p)  for  a  (and  attaching  the  superscripts  ({  in  the  appropriate  places),  and 
ve  end  up  with  simultaneous  bounds  which  apply  (simultaneously)  to  all  p  values 

of  r . 

This  can  be  done  with  any  of  the  four  confidence  interval  formulas  (4.1), 
(3.5),  (5*2),  or  (5. 3), depending  on  our  needs.  For  example,  if  ve  generalize 
(5.3) >  ve  determine  that  100(l-a)°/o  simultaneous  confidence  intervals  for 
d^^(X)  (6.2)  for  all  possible  (j,  j)  and  for  all  %  and  for  all  possible  X 
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are  given  by 


(6.3)  d«>(X)  +  y(r+lHg-l)F 


/«2(f) 


where  (X)  and  are  the  same  functions  of  the  Y^'s  as  d  (X) 

Jj  ©  JK  Jj 

2 

and  S  are  (respectively)  of  the  Y  .  fs  . 
e  jk 

If;  for  the  case  of  p  criterion  variables;  a  region  approach  is  preferred 
to  a  confidence  interval  approach,  this  can  be  arranged  in  a  manner  analogous  to 
that  employed  in  previous  sections.  However,  there  will  be  p  times  as  many 
regions  to  obtain  for  the  p-variate  case  as  there  would  be  for  the  corresponding 
univariate  case. 

Finally,  a  warning  note  should  be  sounded.  As  ovr  techniques  are  adapted  to 
more  and  more  complex  situations  (which  is  the  way  they  have  developed  during  the 
course  of  this  paper),  we  may  in  some  cases  become  faced  with  wider  and  wider 
confidence  intervals  and  with  smaller  and  smaller  regions  of  significance.  In 
fact,  confidence  intervals  might  turn  out  to  be  so  vide  as  to  be  useless,  and 
regions  can  be  so  small  as  to  be  useless.  When  this  sort  of  thing  happens,  it 
means  that  the  sample  sizes  (the  Dj's)  are  too  small  to  furnish  adequate  infor¬ 
mation.  In  general,  it  would  appear  that  this  problem  of  sample  sizes  being  too 
small  might  arise  more  frequently  with  the  more  complicated  types  of  confidence 
intervals  and  regions. 
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